Image processing apparatus, image processing method, and storage medium

ABSTRACT

According to one aspect of the invention, an image processing apparatus comprises: a unit configured to obtain an input image and a subsequent image; a unit configured to obtain N replicated images from the input image; a generating unit configured to generate a low-frequency enhanced image; a subtraction unit configured to generate a high-frequency enhanced image; a synthesizing unit configured to generate a high-frequency output image; and an output unit configured to select and output one of the low-frequency enhanced image and the high-frequency output image. The generating unit comprises a unit configured to obtain a motion vector of an object, and a filter unit configured to apply the filter to pixels around a specified pixel position Q in the replicated image, to obtain a pixel value at a pixel position P in the low-frequency enhanced image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image display technique forimproving a moving image resolution.

2. Description of the Related Art

Recently, commercially available moving image display apparatus fortelevision and the like include so-called liquid crystal displays,plasma displays, and FED displays as well as CRTs. Thus, there arevarious moving image display apparatus. Each type of moving imagedisplay apparatus is designed to reduce motion blurring and flicker byN-folding the frame frequency (frame rate) of an input image signal(that is, dividing one frame into N subframes) and then displaying theresultant image.

The user of a display apparatus designed to emit light almost all thetime in a frame time, for example, a hold-type display apparatus,observes relatively large motion blurring. When the user pursues(follows an area of motion in a moving image with his/her gaze), he/sheobserves larger motion blurring with an increase in the period of lightemission in a frame time. Flicker tends to be observed in synchronismwith frames in a display apparatus, for example, an impulse-type displayapparatus, in which the temporal unevenness of light intensity is largebecause the period of light emission in a frame time is very short.

In contrast to this, when a 60-Hz input image signal is displayed afterthe frame frequency is doubled (N=2, so-called double-speeding) into 120Hz, the period of light emission in one frame is reduced to half, andhence the motion blurring is reduced to about half. In addition, withregard to flicker, doubling the frame frequency into 120 Hz can make thefrequency of flicker synchronized with frames fall out of the range ofthe response characteristics of human vision. This can therefore make itdifficult to observe flicker.

There are two main methods to increase a frame frequency. The firstmethod estimates an image between two frames by detecting the motionvector of an object in an original image. In general, this method iscalled, for example, an intermediate image generating method based onmotion compensation, which is disclosed in, for example, Japanese PatentLaid-Open No. 2004-159294. The first method is expressed as “frame(field) interpolation based on motion compensation”.

According to the second method, performing filter processing for aninput image for each frame will split the image into high spatialfrequency components (high-frequency components) greatly associated withmotion blurring and low spatial frequency components (low-frequencycomponents) greatly associated with flicker. High-frequency componentsare concentrated and displayed on one subframe (one of two double-speedframes corresponding to an original frame). Low-frequency components areconcentrated and displayed on one subframe or distributed and displayedon the two subframes. As the second method, for example, the methodsdisclosed in Japanese Patent Laid-Open Nos. 6-70288 and 2002-351382 andU.S. Patent Laid-Open No. 2006/0227249 each are available. In thisspecification, this second scheme is expressed as a scheme of splittingan image into a plurality of spatial frequency components and displayingthe respective frequency components after distributing them to one or aplurality of subframes, and will be briefly expresses as “spatialfrequency splitting”.

As shown in FIG. 13, the method disclosed in Japanese Patent Laid-OpenNo. 6-70288 temporarily stores an input field image in two fieldmemories while switching the field memories, thus forming twodouble-speed subframes. The speed of an original signal is doubled byalternatively switching these subframes at the rate double the inputfrequency by using a switch SW0. At this time, this method performs theprocessing of suppressing high spatial frequency components for onedouble-speed subframe. As a result, the double-speed subframe havingundergone the processing of suppressing high-frequency components(expressed by “SL” in FIG. 13) contains relatively fewer high-frequencycomponents. The other double-speed subframe (expressed by “SH” in FIG.13) contains relatively more high-frequency components. This makes itpossible to localize high-frequency components on one double-speedsubframe in an output image.

As shown in FIG. 14, according to the method disclosed in JapanesePatent Laid-Open No. 2002-351382, a frame converter doubles the speed ofan input image, and a filter LPF/HPF splits the image into low-frequencycomponents Low and high-frequency components High. In addition, thehigh-frequency components High are multiplied by a gain α for eachdouble-speed subframe. The sign of α is changed for each double-speedsubframe by setting positive α for one double-speed subframe andnegative α for the other double subframe. If it is determined that themotion of an image is large, the absolute value of α may be increased.This makes it possible to localize high-frequency components on onedouble-speed subframe SH.

As shown in FIG. 15, the method disclosed in U.S. Patent Laid-Open No.2006/0227249 generates high-frequency component data H by applying afilter HPF to an input image. In addition, adding the high-frequencycomponent data H to the input image will generate high-frequency outputimage data SH. Subtracting the high-frequency component data H from aninput image A will generate low-frequency image data SL. Switching thesedata at a frequency double the frame frequency of the input image usingthe switch SW0 can output a double-speed image one of whose subframe haslocalized high-frequency components.

The first method “frame (field) interpolation based on motioncompensation” has the problem that since it requires the processing ofgenerating an intermediate image by using the image data of adjacentframes, a large amount of calculation is required. The second method“spatial frequency splitting” has the problem that since displayedimages of the first and second subframes do not properly reflect thedifference of each time to be displayed, an image lag (tail-blurring)occurs in an area with motion.

SUMMARY OF THE INVENTION

According to the present invention, it is possible to reduce an imagelag in an area with motion which is caused when an image is displayed ata double speed by using conventional “spatial frequency splitting”, witha smaller amount of calculation than that required by conventional“frame (field) interpolation based on motion compensation”.

According to one aspect of the invention, an image processing apparatuswhich processes a moving image comprising successive frame imagescomprises: a unit configured to obtain a frame image of interest as aninput image, and to obtain a frame image subsequent to the frame imageof interest as a subsequent image; a unit configured to obtain Nreplicated images from the input image; a generating unit configured togenerate a low-frequency enhanced image whose low-frequency component isenhanced, from the replicated image by using a filter for enhancing alow-frequency component; a subtraction unit configured to generate ahigh-frequency enhanced image whose high-frequency component isenhanced, from the low-frequency enhanced image and the replicatedimage; a synthesizing unit configured to generate a high-frequencyoutput image by synthesizing the high-frequency enhanced image and thelow-frequency enhanced image at a predetermined ratio; and an outputunit configured to select and output one of the low-frequency enhancedimage and the high-frequency output image in place of each of thereplicated images, the generating unit comprising a unit configured toobtain a motion vector of an object depicted in the input image and thesubsequent image, and a filter unit configured to generate thelow-frequency enhanced image, when processing a pixel at a pixelposition P in the replicated image, by performing, for each pixelposition in the replicated image, a process of specifying a pixelposition Q spaced apart from the pixel position P in the replicatedimage by a predetermined distance in an opposite direction to the motionvector, a process of applying the filter to pixels around the specifiedpixel position Q in the replicated image, to obtain a pixel value at thepixel position P in the low-frequency enhanced image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the arrangement of an image displayapparatus according to the first embodiment;

FIG. 2 is a view for explaining an example of a filter function used inthe first embodiment;

FIG. 3 is a view showing an example of filter coefficients used in thefirst embodiment;

FIG. 4 is a view showing an example of a double-speed (N=2) subframeimage display sequence in the first embodiment;

FIG. 5 is a graph for explaining a motion vector used in the firstembodiment;

FIG. 6 is a view showing an example of filter coefficients used in thefirst embodiment;

FIG. 7 is a view showing an example of filter coefficients used in thefirst embodiment;

FIG. 8 is a view showing an example of filter coefficients used in thefirst embodiment;

FIG. 9 is a view showing an example of filter coefficients used in thefirst embodiment;

FIG. 10 is a flowchart showing processing in the first embodiment;

FIG. 11 is a view for explaining effects in the first embodiment;

FIG. 12 is a block diagram showing the arrangement of an image displayapparatus according to the third embodiment;

FIG. 13 is a block diagram schematically showing a conventional imagedisplay apparatus;

FIG. 14 is a block diagram schematically showing a conventional imagedisplay apparatus;

FIG. 15 is a block diagram schematically showing a conventional imagedisplay apparatus; and

FIG. 16 is a block diagram showing the electrical arrangement of acomputer according to the fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

The embodiments of the present invention will be described below withreference to the accompanying drawings. However, the scope of thepresent invention is not limited to the following embodiments.

First Embodiment

FIG. 1 is a block diagram showing the arrangement of an image displayapparatus according to the first embodiment. Referring to FIG. 1,reference numeral 100 denotes a frame rate converting unit; 101, alow-pass filter (to be referred to as an LPF hereafter); 102, asubtracter; 103, a multiplier; 104, an adder; 105, a switch; 106, amotion vector detecting unit; 107, a filter coefficient generating unit;and 108, an output unit.

The frame rate converting unit 100 acquires moving image data in oneframe unit, and generates N subframe images from one frame image. Morespecifically, the frame rate converting unit 100 includes a frame memory(not shown), and stores input moving image data as an input image in theframe memory on a frame basis. The frame rate converting unit 100 thengenerates N subframe images as replicated images by reading one frameimage of interest (input image) N times at the N times higher frequencythan input video data. Assume that in this case, one subframe imagegenerated by the frame rate converting unit 100 is denoted by referencesymbol A. The frame rate converting unit 100 sequentially outputs thesubframe image A to the LPF 101, the subtracter 102, and the adder 104.A coefficient N which defines the number of subframe images may bedetermined in advance and set to the apparatus according to thisembodiment, or may be acquired from outside the apparatus.

The LPF 101 is a two-dimensional low-pass filter. The LPF 101 cuts offthe upper limit spatial frequency from the subframe image A generated bythe frame rate converting unit 100 by processing the image using afilter, thereby generating a low-frequency enhanced image L whoselow-frequency components are enhanced. The LPF 101 acquires filtercoefficients used by itself from the filter coefficient generating unit107.

This embodiment uses a filter function based on a Gaussian function as alow-pass filter. However, the type of filter as a base is notspecifically limited. For example, it is possible to use a Gaussianfunction or a filter which provides a moving average or a weightedmoving average. FIG. 2 is a view for explaining an example of the filterfunction used by the LPF 101. FIG. 2 schematically shows the filterfunction generated based on a Gaussian function. The filter function inFIG. 2 plots a normal distribution in which the center of the weight isplaced on the reference point.

The operation of the LPF 101 will be described in detail below. FIG. 3shows an example of the filter coefficients used by the LPF 101.Referring to FIG. 3, the filter size (so-called kernel size) is 7×7pixels. Note, however, that the filter size and filter coefficients aremerely examples, and any filter size and filter coefficients may beused. In this case, filter coefficients are so-called kernel valuesbefore normalization.

Filter processing for one pixel of interest in a subframe image will bedescribed first. Assume that in this case, a pixel value A ofcoordinates (x, y) in the i:th subframe image generated from one frameimage is represented by Ai(x, y). First of all, the coordinates of apixel of interest are represented by Ai(0, 0). The LPF 101 performsfilter computation for a subframe image by using the filter coefficientsshown in FIG. 3. In an example using the filter coefficients in FIG. 3,LPF(Ai(0, 0)) as an LPF output value of the pixel of interest Ai(0, 0)is determined based on equation (1):

$\begin{matrix}{{{LPF}\left( A_{n{({0,0})}} \right)} = {\frac{1}{256}\begin{pmatrix}0 & 1 & 2 & 4 & 2 & 1 & 0 \\1 & 2 & 6 & 8 & 6 & 2 & 1 \\2 & 6 & 8 & 16 & 8 & 6 & 2 \\4 & 8 & 16 & 32 & 16 & 8 & 4 \\2 & 6 & 8 & 16 & 8 & 6 & 2 \\1 & 2 & 6 & 8 & 6 & 2 & 1 \\0 & 1 & 2 & 4 & 2 & 1 & 0\end{pmatrix}\begin{pmatrix}A_{n{({{- 3},{- 3}})}} & A_{n{({{- 2},{- 3}})}} & A_{n{({{- 1},{- 3}})}} & A_{n{({0,{- 3}})}} & A_{n{({1,{- 3}})}} & A_{n{({2,{- 3}})}} & A_{n{({3,{- 3}})}} \\A_{n{({{- 3},{- 2}})}} & A_{n{({{- 2},{- 2}})}} & A_{n{({{- 1},{- 2}})}} & A_{n{({0,{- 2}})}} & A_{n{({1,{- 2}})}} & A_{n{({2,{- 2}})}} & A_{n{({3,{- 2}})}} \\A_{n{({{- 3},{- 1}})}} & A_{n{({{- 2},{- 1}})}} & A_{n{({{- 1},{- 1}})}} & A_{n{({0,{- 1}})}} & A_{n{({1,{- 1}})}} & A_{n{({2,{- 1}})}} & A_{n{({3,{- 1}})}} \\A_{n{({{- 3},0})}} & A_{n{({{- 2},0})}} & A_{n{({{- 1},0})}} & A_{n{({0,0})}} & A_{n{({1,0})}} & A_{n{({2,0})}} & A_{n{({3,0})}} \\A_{n{({{- 3},1})}} & A_{n{({{- 2},1})}} & A_{n{({{- 1},1})}} & A_{n{({0,1})}} & A_{n{({1,1})}} & A_{n{({2,1})}} & A_{n{({3,1})}} \\A_{n{({{- 3},2})}} & A_{n{({{- 2},2})}} & A_{n{({{- 1},2})}} & A_{n{({0,2})}} & A_{n{({1,2})}} & A_{n{({2,2})}} & A_{n{({3,2})}} \\A_{n{({{- 3},3})}} & A_{n{({{- 2},3})}} & A_{n{({{- 1},3})}} & A_{n{({0,3})}} & A_{n{({1,3})}} & A_{n{({2,3})}} & A_{n{({3,3})}}\end{pmatrix}}} & (1)\end{matrix}$

The respective pixels constituting one subframe image are sequentiallyset as pixels of interest and processed according to equation (1),thereby sequentially obtaining LPF output values for each pixel ofinterest. The image obtained by updating the pixel values of interest ofthe subframe image with the LPF output values is acquired as alow-frequency enhanced image. In this case, the pixels of interest maybe all the pixels of the subframe image or may be some pixels of thesubframe image. This embodiment performs filter processing for all thepixels of a subframe image. Therefore, in the processing in which themotion vector detecting unit 106 and the filter coefficient generatingunit 107 generate filter coefficients, the motion vector detecting unit106 and the filter coefficient generating unit 107 also set all thepixels of an image sequentially as pixels of interest. When the LPF 101is to perform filter processing for some pixels of an image, the motionvector detecting unit 106 and the filter coefficient generating unit 107may set a portion of the image as a portion of interest.

The motion vector detecting unit 106 acquires an input image from theframe rate converting unit 100. The motion vector detecting unit 106acquires the image of a frame adjacent and subsequent to the inputimage, stored in the frame rate converting unit 100, as a subsequentimage. The motion vector detecting unit 106 obtains the motion vector ofan object depicted in the input image and the subsequent image, from theinput image and the subsequent image. Finally, the motion vectordetecting unit 106 estimates a motion vector corresponding to the inputimage for each pixel of the input image. The motion vector detectingunit 106 outputs the motion vector corresponding to each pixel of theinput image to the filter coefficient generating unit 107.

The filter coefficient generating unit 107 generates a filtercoefficient for each pixel of the input image by using a motion vector.The filter coefficient generating unit 107 outputs the generated filtercoefficient to the LPF 101. In this embodiment, the filter coefficientgenerating unit 107 acquires filter coefficients by moving a filterfunction. This processing allows the filter to be applied to pixelsaround a pixel position Q, where the position Q is spaced apart from apixel position P in the input image which corresponds to a pixelposition P of interest in the replicated image by a predetermineddistance in the direction opposite to the motion vector. The motionvector detecting unit 106 and the filter coefficient generating unit 107will be described in detail later.

The subtracter 102 subtracts each pixel value of the low-frequencyenhanced image L generated by the LPF 101 from each pixel value of theinput image according to equation (2). In this manner, the subtracter102 newly generates a high-frequency enhanced image H whosehigh-frequency components are enhanced.H=A−L  (2)

The multiplier 103 determines the amount of high-frequency components tobe distributed to each subframe image. More specifically, the multiplier103 generates a high frequency distribution image DH by multiplying eachpixel value of the high-frequency enhanced image H by (N−a)/a. Themultiplier 103 then outputs the high frequency distribution image DH tothe adder 104. As described above, the coefficient N is the number ofsubframe images generated from one frame image. The coefficient aindicates the number of subframe images in which the high-frequencyenhanced images H are located. That is, the apparatus according to thisembodiment outputs a high-frequency enhanced images which correspond toa one-frame input image. The coefficient a may be determined in advanceand set to the apparatus according to this embodiment or may beexternally acquired.

The adder 104 adds each pixel value of the high frequency distributionimage DH generated by the multiplier 103 to each pixel value of theinput image. In this manner, the adder 104 generates a newhigh-frequency output image SH. The adder 104 performs computationaccording to equation (3):SH=A+{(N−a)/a}H=L+(N/a)H  (3)

As a result of the processing performed by the multiplier 103 and theadder 104, the high-frequency output image SH output from the adder 104becomes an image obtained by adding a high-frequency enhanced image to alow-frequency enhanced image at a predetermined ratio (N/a).

The switch 105 selects either the low-frequency enhanced image Lgenerated by the LPF 101 or the high-frequency output image SH generatedby the adder 104. The switch 105 outputs the selected image in place ofthe replicated image. FIG. 4 is a view for explaining the displaysequence of subframe images in double speed (N=2) in this embodiment. Inthe example in FIG. 4, the frame image of the m:th frame Fm is convertedinto two (N=2) subframe images. In addition, in the example in FIG. 4,the high-frequency enhanced image H is located in one subframe image(a=1).

As shown in FIG. 4, the switch 105 outputs a high-frequency output imageSHm and a low-frequency enhanced image Lm as the first and secondsubframe images, respectively. A method by which the switch 105 selectseither the low-frequency enhanced image L or the high-frequency outputimage SH is not specifically limited. The switch 105 may selectlow-frequency enhanced images L and high-frequency output images SH soas to make their output intervals become uniform. Alternatively, theswitch 105 may select high-frequency output images SH a timesconsecutively, and then select low-frequency enhanced images L (N−a)times consecutively.

The motion vector detecting unit 106 and the filter coefficientgenerating unit 107 will be described in detail below. The motion vectordetecting unit 106 estimates how much an object depicted in an m:thframe image Fm has moved on an (m+1)th frame image Fm+1. The motionvector detecting unit 106 acquires a motion vector concerning each pixelconstituting the frame image Fm.

FIG. 5 is a graph for explaining a motion vector according to thisembodiment. FIG. 5 shows an image of 11×11 pixels centered on a pixel ofinterest (represented by P in FIG. 5) in the m:th frame image Fm. Whenthe motion vector detecting unit 106 estimates that the pixel ofinterest (represented by P in FIG. 5) has moved to the coordinates ((x,y)=(3, −4)) of Q shown in FIG. 5 in the (m+1)th input frame image Fm+1,a motion vector V(x, y) of P is (+3, −4). Vx=+3 represents a motionvector in the x direction; and Vy=−4, a motion vector in the ydirection.

When Vx has the sign “+”, Vx indicates movement in the x-axis positivedirection in FIG. 5. When Vx has the sign “−”, Vx indicates movement inthe x-axis negative direction in FIG. 5. When Vy has the sign “+”, Vyindicates movement in the y-axis positive direction in FIG. 5. When Vyhas the sign “−”, Vy indicates movement in the y-axis negative directionin FIG. 5. The absolute value of each numerical value indicates a movingamount (the number of pixels of movement).

When the motion vector detecting unit 106 estimates that an objectindicated by a pixel of interest has no movement, the motion vectordetecting unit 106 outputs motion vector V(x, y)=(0, 0). Likewise, whena scene change occurs between frames, motion vector detecting unit 106can output motion vector V(x, y)=(0, 0). The definition of the motionvector V(x, y) in this case is merely an example. A motion vector mayindicate an approximate direction and distance (the number of pixels) inand by which a pixel of interest has moved between input frames.

The motion vector detecting unit 106 sequentially sets each pixel of theframe image Fm as a pixel of interest and sequentially acquires themotion vector of each pixel of interest. In the description of thisembodiment, in order to comprehensibly express the relationship betweena motion vector and coordinate information, a coordinate system issequentially converted such that the coordinates of a pixel of interestare represented by ((x, y)=(0, 0)). The filter coefficient generatingunit 107 acquires the motion vector V(x, y) obtained by the motionvector detecting unit 106.

The filter coefficient generating unit 107 generates filter coefficientsused by the LPF 101 in filter computation. The filter coefficientgenerating unit 107 outputs the generated filter coefficients to the LPF101. The filter coefficient generating unit 107 generates filtercoefficients based on the motion vector V(x, y) obtained by the motionvector detecting unit 106, the number N of subframe images for one frameimage, and a number i of a subframe image subjected to filtercomputation. The filter coefficient generating unit 107 prepares onenumerical value set used by the LPF 101 for each pixel contained in asubframe image. That is, the filter coefficient generating unit 107prepares numerical value sets equal in number to the number of pixels ofa subframe image. The filter coefficient generating unit 107 outputs theprepared numerical sets as filter coefficients. This processing will bedescribed in detail below.

The subframe image numbers i are the numbers assigned to subframe imagesin the order of display. For example, in double-speeding (N=2), as shownin FIG. 4, the subframe image to be displayed first is represented byi=1, and the subframe image to be displayed second is represented by i=2(1≦i≦N). As described above for example, the values of the filterfunction generated based on a Gaussian function form a normaldistribution in which the center of the weight is placed on thereference point. In this embodiment, the filter coefficient generatingunit 107 determines a reference point based on the motion vector (x, y),the subframe image count N, and the subframe image number i.

In this case, the filter coefficient generating unit 107 sets one pixelof a subframe image as a pixel of interest. Based on equations (4), thefilter coefficient generating unit 107 then calculates coordinates (X,Y) of a reference point of the filter function applied to the pixel ofinterest.X=(i−1)×((−Vx/N)Y=(i−1)×((−Vy/N)  (4)

At this time, the coordinates (0, 0) are the central coordinates of thefilter function. Let Vx be the x component of the motion vector V(x, y),and Vy be the y component of the vector.

For example, when motion vector V(x, y)=(+3, −6) and N=2, thecoordinates of the reference point of the first subframe image (i=1) aregiven as coordinates (X, Y)=(0, 0), and the coordinates of the referencepoint of the second subframe image (i=2) are given as coordinates (X,Y)=(−1.5, 3) according to equations (4). In addition, for example, whenmotion vector V(x, y)=(+3, −6) and N=3, the coordinates of the referencepoint of the first subframe image (i=1) are given as coordinates (X,Y)=(0, 0), the coordinates of the reference point of the second subframeimage (i=2) are given as coordinates (X, Y)=(−1, 2), and the coordinatesof the reference point of the third subframe image (i=3) are given ascoordinates (X, Y)=(−2, 4).

A case in which motion vector V(x, y)=(+2, +2) and N=2 will be describedin more detail. With the application of equations (4), the coordinatesof the reference point of the first subframe image (i=1) are given ascoordinates (X, Y)=(0, 0), and the coordinates of the reference point ofthe second subframe image (i=2) are given as coordinates (X, Y)=(−1,−1). A method of calculating filter coefficients used for the first andsecond subframe images will be described below with reference to FIGS. 6to 9.

FIG. 6 shows an example of a filter function according to thisembodiment. FIG. 6 schematically shows the filter function to be appliedto the first subframe image (i=1) when motion vector V(x, y)=(+2, +2)and N=2. FIG. 7 shows an example of filter coefficients in thisembodiment. FIG. 7 shows the filter coefficients for the first subframeimage (i=1) when motion vector V(x, y)=(+2, +2) and N=2.

As shown in FIGS. 6 and 7, the filter coefficient generating unit 107generates filter coefficients for the first subframe image according toequations (4) such that the coordinates (X, Y) of the reference pointbecome (0, 0). The filter coefficient generating unit 107 outputs thegenerated filter coefficients to the LPF 101. This embodiment may use,as filter coefficients, the coefficient values at the respectivecoordinates obtained by substituting the respective coordinates offilter coefficients into a two-dimensional Gaussian function held inadvance in the apparatus or acquired externally.

The filter coefficient generating unit 107 obtains filter coefficientsby repeating the above processing using the motion vectors estimated bythe motion vector detecting unit 106 for the respective pixels of theinput image. That is, the filter coefficient generating unit 107 obtainsfilter coefficients for the respective pixels of each subframe image.The LPF 101 acquires the filter coefficients for the first subframeimage from the filter coefficient generating unit 107. The LPF 101generates the low-frequency enhanced image L by sequentially performingfilter computation for the respective pixels of the first subframe image(i=1) using the acquired filter coefficients.

FIG. 8 shows an example of a filter function in this embodiment. FIG. 8schematically shows the filter function for the second subframe image(i=2) when motion vector V(x, y)=(+2, +2) and N=2. FIG. 9 shows anexample of filter coefficients in this embodiment. FIG. 9 shows filtercoefficients for the second subframe image (i=2) when motion vector V(x,y)=(+2, +2) and N=2.

As shown in FIGS. 8 and 9, the filter coefficient generating unit 107generates filter coefficients for the second subframe image according toequations (4) such that the coordinates (X, Y) of the reference pointbecome (−1, −1). The filter coefficient generating unit 107 then outputsthe generated filter coefficients to the LPF 101. In this embodiment,the filter coefficient generating unit 107 translates thetwo-dimensional Gaussian function, held in advance in the apparatus oracquired externally, by −1 in the X-axis direction and by −1 in theY-axis direction. The filter coefficient generating unit 107 may use, asfilter coefficients, the coefficient values obtained by substituting therespective coordinates of filter coefficients into the translatedGaussian function.

The filter coefficient generating unit 107 obtains filter coefficientsby repeating the above processing using the motion vectors estimated bythe motion vector detecting unit 106 for the respective pixels of theinput image, as in the case of the first subframe image. That is, thefilter coefficient generating unit 107 obtains filter coefficients forthe respective pixels of the input image and the second subframe imageobtained by replicating the input image. The LPF 101 acquires the filtercoefficients for the second subframe image from the filter coefficientgenerating unit 107. The LPF 101 generates the low-frequency enhancedimage L by sequentially performing filter computation for the respectivepixels of the second subframe image (i=2) using the acquired filtercoefficients.

Processing in the first embodiment will be described below withreference to the flowchart of FIG. 10. First of all, the frame rateconverting unit 100 acquires a frame image contained in moving imagedata as an input image (step S1001). The frame rate converting unit 100then acquires the image of the frame following the input image as asubsequent image (step S1002).

The motion vector detecting unit 106 acquires the input image and thesubsequent image from the frame rate converting unit 100. The motionvector detecting unit 106 then detects the moving amount of an objectindicated by each pixel of the input image between the input image andthe subsequent image. The motion vector detecting unit 106 generates themoving amount as a motion vector V(x, y) concerning each pixel of theinput image (step S1003). In the processing in step S1003, the motionvector detecting unit 106 may estimate a moving amount by using aplurality of frame images or estimate a moving amount by using subframeimages.

Subsequently, the frame rate converting unit 100 reads out the acquiredinput image at a clock (frequency) N times that of moving image data.The frame rate converting unit 100 outputs the readout image as asubframe image together with a subframe number i (step S1004). Thefilter coefficient generating unit 107 generates filter coefficientsused for filter computation by the LPF 101 based on the motion vectorV(x, y), the subframe image count N, and the subframe image number i(step S1005). The subframe image count N may be stored in the apparatusin advance or input externally. The subframe image number i is acquiredfrom the frame rate converting unit 100.

The LPF 101 sets one pixel in the subframe image output from the framerate converting unit 100 as a pixel of interest (step S1006). The LPF101 performs filter computation for the pixel of interest by using thefilter coefficient acquired from the filter coefficient generating unit107, and acquire a converted the pixel value (step S1007). The LPF 101then determines whether it has set all the pixels of the subframe imageoutput from the frame rate converting unit 100 as pixels of interest instep S1006 (step S1008). If the LPF 101 has not set all the pixels aspixels of interest, the process returns to step S1006, in which the LPF101 sets the next pixel as a pixel of interest. If the LPF 101 has setall the pixels as pixels of interest, the process advances to stepS1009. With the processing in steps S1006 to S1008, the LPF 101generates the low-frequency enhanced image L by performing filtercomputation and pixel value conversion.

The subtracter 102 acquires the low-frequency enhanced image L from theLPF 101. The subtracter 102 then generates the high-frequency enhancedimage H according to equation (2) (step S1009). The multiplier 103multiplies each pixel value of the high-frequency enhanced image Hacquired from the subtracter 102 by ((N−a)/a). The multiplier 103 thenoutputs the image having undergone the multiplication processing (stepS1010).

The adder 104 acquires the image processed by the multiplier 103. Theadder 104 acquires the subframe image output from the frame rateconverting unit 100 in step S1004. The adder 104 then generates thehigh-frequency output image SH whose high-frequency components areenhanced according to equation (3) (step S1011).

The switch 105 determines to select the high-frequency output image orthe low-frequency enhanced image (step S1012). The determinationcriterion to be set is not specifically limited, as described above. Theswitch 105 may alternatively select the low-frequency enhanced image andthe high-frequency output image. Alternatively, the switch 105 mayconsecutively select a high-frequency output images, and thenconsecutively select (N−a) low-frequency enhanced images. The switch 105may select a high-frequency output images and (N−a) low-frequencyenhanced images during repetitive execution of steps S1004 to S1015. Ifthe switch 105 selects a low-frequency enhanced image in step S1012, theswitch 105 acquires the low-frequency enhanced image L from the LPF 101and outputs it (step S1013). If the switch 105 selects a high-frequencyoutput image in step S1012, the switch 105 acquires the high-frequencyoutput image SH from the adder 104 and outputs it (step S1014).

Finally, the frame rate converting unit 100 determines whether Nsubframe images of the acquired image have been output (step S1015). IfN subframe images have not been output, the process returns to stepS1004. If N subframe images have been output, the frame rate convertingunit 100 terminates the processing for the acquired image.

As described above, the filter coefficient generating unit 107 generatesfilter coefficients for filter processing for the second and subsequentsubframe images by moving the reference point of the filter function inthe exact opposite direction to the detected moving direction. Usingthese filter coefficients can reduce the image lag in an area withmotion which occurs in the second method “spatial frequency splitting”.

FIG. 11 is a view for explaining an effect in this embodiment. In FIG.11, 1101 schematically shows a movement of an image formed of thenumbers “23456789” to the right (in the x direction) between frames bytwo pixels. For the sake of descriptive convenience, assume that eachpixel represents one number. Pixel numbers (II to X) are assigned to therespective pixels. In FIG. 11, 1102 indicates each subframe imageobtained by doubling the speed (N=2) of each frame image in 1101 in FIG.11 by frame rate conversion processing. In FIG. 11, 1102 indicates anexample of the filter function generated based on this embodiment whenthe grayed pixels are set as pixels of interest. FIG. 11 illustratesone-dimensional images for the sake of simplicity.

Referring to FIG. 11, the motion vector V(x, y) of the pixel of interest(pixel number VI) between a frame image m and a frame image m+1 is (+2,0). In this case, with the application of equations (4), the referencepoint of the first subframe image (i=1) has coordinates (X, Y)=(0, 0),and the reference point of the second subframe image (i=2) hascoordinates (X, Y)=(−1, 0).

The filter coefficient to be applied to the pixel number VI of the firstsubframe image of the frame image m is obtained from a filter functionwith a weight being assigned to the pixel value corresponding to thepixel number VI. The filter coefficient to be applied to the pixelnumber VI of the second subframe image of the frame image m is obtainedfrom a filter function with a weight being assigned to the pixel valuecorresponding to the pixel number V. In addition, the filter coefficientto be applied to the pixel number VI of the first subframe image of theframe image m+1 is obtained from a filter function with a weight beingassigned to the pixel number VI because (X, Y)=(0, 0) according toequations (4).

The second subframe image of the frame image m is displayed between thefirst subframe image of the frame image m and the first subframe imageof the frame image m+1. For this reason, the second subframe image ofthe frame image m may be an intermediate image between the firstsubframe image of the frame image m and the first subframe image of theframe image m+1.

Consider a case in which a low-frequency enhanced image is generated byapplying the same filter function as that for the first subframe imageto the second subframe image. A low-frequency enhanced image used togenerate the second subframe image of the frame m is the same as thatused to generate the first subframe image of the frame m. That is, thesame image as the first subframe image of the frame m is displayed againbetween the frame m and the frame m+1 with a time lag, although theyhave different enhanced frequency components. For this reason, thesecond subframe image of the frame m is visually perceived as an imagelag of the first subframe image of the frame m.

For example, a case in which filter processing is performed for thepixels corresponding to the pixel number VI in FIG. 11 will be describedwith reference to FIG. 11. A filter function with weights being assignedto the pixel values corresponding to the pixel number VI is used for thefirst and second subframe images of the frame image m. Therefore,regarding both the first and second subframe images of the frame imagem, pixel values with weights being assigned to the numerical value “6”are generated by filter processing, thereby generating the low-frequencyenhanced image L. A filter function with a weight being assigned to thepixel number VI is also used for the first subframe image of the frameimage m+1. Regarding the first subframe image of the frame image m+1 aswell, the low-frequency enhanced image L is generated by generating apixel value with a weight being assigned to the numerical value “4” byfilter processing. However, it would be better to perform filterprocessing for the second subframe image of the frame image m, displayedbetween the frame m and the frame m+1, with a weight being assigned to anumerical value existing between the numerical value “6” and thenumerical value “4”.

According to this embodiment, regarding the second subframe image of theframe m, it is possible to perform filter processing for the pixel ofinterest by using a filter function with a weight being assigned to thepixel value corresponding to the pixel number V, in other words, thenumerical value “5”. That is, it is possible to reduce the image lagadaptively in accordance with motion. According to the second method“spatial frequency splitting”, high-frequency components greatlyassociated with motion blurring are concentrated on one of a pluralityof subframes. In addition, this method distributes low-frequencycomponents which are not greatly associated with motion blurring toother subframe, or to a plurality of subframes, and displays them. Inthis manner, “spatial frequency splitting” can reduce motion blurring ina moving image recognized by a viewer. In conventional “spatialfrequency splitting”, however, since each subframe does not reflect thedifference of each time to be displayed, an image lag occurs in an areawith motion. As described above, according to this embodiment, when thefrequency components of the respective subframes are obtained, a motionvector is referred to. This makes it possible to generate subframes eachproperly reflecting the difference of each time to be displayed. Thisembodiment can therefore reduce an image lag in an area with motionwhich occurs in the second method “spatial frequency splitting”.

In this embodiment, in particular, low-frequency enhanced images aremade to reflect the difference of each time to be displayed. Therefore,regarding a plurality of subframes generated from one frame, a betterresult is obtained when many high-frequency enhanced images(high-frequency output images) are selected as subframes to be displayedearlier, and many low-frequency enhanced images are selected assubframes to be displayed later. For example, in double-speed display,it is possible to display a high-frequency output image as the firstsubframe and a low-frequency enhanced image as the second subframe. Whendisplaying in four-times higher frequency than an input moving image, itis possible to display a high-frequency output image as the firstsubframe and low-frequency enhanced images as the second to fourthsubframes or to display high-frequency output images as the first andsecond subframes and low-frequency enhanced images as the third andfourth sub frames.

According to the method of this embodiment, after the LPF 101 performsfilter processing while changing filter coefficients, it is possible togenerate subframe images by using only simple addition, subtraction, andmultiplication. For this reason, unlike the first method “frame (field)interpolation based on motion compensation”, it is not necessary togenerate an intermediate image by using the image data of adjacentframes. Therefore, the method of this embodiment can generate subframeimages with a small amount of calculation.

<Modification>

In the first embodiment, the filter coefficient generating unit 107generates filter coefficients. In addition, the LPF 101 performs filterprocessing based on the generated filter coefficients. The filtercoefficient generating unit 107 acquires filter coefficients by moving afilter function. Using these filter coefficients allows the filter to beapplied to pixels around the pixel position Q, where the position Q isspaced apart from the pixel position P in the input image whichcorresponds to the pixel position P of interest in a replicated image bya predetermined distance in the direction opposite to the motion vector.

However, using the pixel position P of the input image which correspondsto the pixel position P of interest in the subframe image (replicatedimage) can specify the pixel position Q spaced apart by a predetermineddistance in the opposite direction to the motion vector. It is possibleto apply the filter to pixels around the specified pixel position Qwithout using the filter coefficients generated by the filtercoefficient generating unit 107. That is, this modification uses thecoordinates calculated as the coordinates of a reference point of thefilter function as the center coordinates of pixels to which the filteris applied.

This modification will be described in detail below. Although thearrangement of the modification is almost the same as that of the firstembodiment, the filter coefficient generating unit 107 does not exist. Amotion vector detecting unit directly inputs a motion vector to the LPF101. The LPF 101 changes the application range of the filter inaccordance with the motion vector. The LPF 101 calculates thecoordinates of a reference point according to equations (4) like thefilter coefficient generating unit 107 in the first embodiment. Thefirst embodiment uses a calculated reference point as a reference pointof a filter function. However, this modification uses a calculatedreference point to obtain the center point of a pixel group to which thefilter is applied.

The LPF 101 sets one pixel of a subframe image to be processed as apixel of interest. This pixel position is represented by P. The LPF 101then specifies, as the coordinate position Q, the position of areference point calculated when the pixel position P of the input imagewhich corresponds to the pixel position P of the subframe image is setas an origin point. If, for example, the coordinates (X, Y) of areference point are (−1, 2), the LPF 101 specifies, as the coordinatepoint Q, the point which is moved from the pixel position P by 2 in theY-axis negative direction and by 1 in the X-axis positive direction. TheLPF 101 then applies the filter to pixels around the coordinate positionQ. This modification does not use the filter coefficient generating unitwhich generates filter coefficients by moving a filter function. Insteadof this unit, the modification applies the same filter to differentpositions of interest. More specifically, the LPF 101 determines an LPFoutput value according to equation (1). At this time, the LPF 101obtains an LPF output value such that the coordinates of the coordinateposition Q become An(0, 0). The LPF 101 sets the obtained LPF outputvalue as the converted pixel value of the pixel of interest (pixelposition P) of the subframe image. The LPF 101 can obtain alow-frequency enhanced image by performing this processing for eachpixel of the subframe image.

Second Embodiment

In the first embodiment, the filter coefficient generating unit 107calculates the coordinates (X, Y) of a reference point according toequations (4). However, the way of obtaining a reference point is notlimited to this method. In some cases, the absolute value of eachcomponent of a motion vector V(x, y) greatly exceeds the filter size(kernel size) of the filter function. Assume that the absolute value ofeach component of a motion vector is very large. In this case, when areference point moves, the filter shape may become unbalanced because,for example, the maximum size of the filter size (kernel size) islimited on a circuit. That the filter shape becomes unbalanced is that,for example, the filter size becomes asymmetric about the center of thefilter function in the x direction. It is possible to use the asymmetricfunction. However, it is also possible to impose a limitation on themoving amount of a reference point.

In the second embodiment, a limitation may be imposed on the movingamount of a reference point by any method as long as the absolute valueof the moving amount of the reference point is smaller than that in thefirst embodiment. That is, the second embodiment may determine a movingamount according to inequalities (5):|X|≦|(i−1)×((−Vx)/N)||Y|≦|(i−1)×((−Vy)/N)|  (5)

At this time, the coordinates (0, 0) are the center coordinates of thefilter function. Let Vx be the x component of a motion vector V(x, y),and Vy be the y component of the motion vector V(x, y). |X|, |Y|,|(i−1)|×((−Vx)/N)| and |(i−1)|×((−Vy)/N)| indicate the absolute valuesof X, Y, ((i−1)×((−Vx)/N)) and ((i−1)×((−Vy)/N)).

For example, the coordinate value obtained according to equations (4)may be simply reduced to half. Alternatively, it is possible to obtainthe square root of the absolute value of the coordinate value obtainedaccording to equations (4). According to another method, it is possibleto improve the balance of the filter shape by reducing the filter size.For example, after limiting the moving amount of a reference pointaccording to inequalities (5), the filter function may be adaptivelychanged. For example, the variance of the filter function may be changedin accordance with the filter size on the circuit.

Setting a reference point according to, for example, inequalities (5) asin this embodiment instead of applying equations (4) makes it possibleto reduce an image lag in an area with motion which occurs in the secondmethod “spatial frequency splitting” as in the first embodiment. It isnot necessary to generate an intermediate image by using the image dataof adjacent frames as in the first method “frame (field) interpolationbased on motion compensation”; it is possible to generate a subframeimage with a small amount of calculation as in the first embodiment.

Third Embodiment

In the first and second embodiments, the LPF 101 performs spatialfrequency component splitting. In the third embodiment, a high-passfilter (to be referred to as an HPF hereinafter) performs spatialfrequency component splitting. An HPF may be, for example, a filterfunction represented by a Laplacian filter which determines filtercoefficients based on a spatial second derivative.

FIG. 12 is a block diagram showing the arrangement of an image displayapparatus according to the third embodiment. The same reference numeralsas in FIG. 1 denote the same constituent elements as those in the firstembodiment. In the third embodiment, the operations of a frame rateconverting unit 100, motion vector detecting unit 106, and output unit108 are the same as those in the first embodiment.

Unlike in the first embodiment, a filter coefficient generating unit 107outputs filter coefficient used by an HPF 1201. The filter coefficientgenerating unit 107 obtains filter coefficients based on a basic filterfunction designed to cut off (filter) lower frequencies. However, as inthe first embodiment, the filter coefficient generating unit 107 obtainsfilter coefficients by translating the basic filter function inaccordance with the motion vector acquired from the motion vectordetecting unit 106. More specifically, the filter coefficient generatingunit 107 calculates coordinates (X, Y) of a reference point of thefilter function according to equations (4) in the first embodiment orinequalities (5) in the second embodiment. The filter coefficientgenerating unit 107 then obtains filter coefficients by translating thebasic filter function in accordance with the coordinates of thereference point.

An HPF 1201 (alternative generating unit) cuts off (filters) lowerspatial frequencies, than a limit frequency, of a subframe image Aacquired from the frame rate converting unit 100 by using the filtercoefficients generated by the filter coefficient generating unit 107. Inthis manner, the HPF 1201 generates a high-frequency enhanced image H.Reference numeral 1202 denotes a subtracter, which subtracts thehigh-frequency enhanced image H generated by the HPF 1201 from thesubframe image A according to equation (6). In this manner, thesubtracter 1202 calculates a low-frequency enhanced image L.L=A−H  (6)

The operations of an multiplier 103, adder 104, and switch 105 using thelow-frequency enhanced image L and the high-frequency enhanced image Hare the same as those in the first embodiment.

Like the first and second embodiments, this embodiment can reduce animage lag in an area with motion which occurs in the second method“spatial frequency splitting”. It is not necessary to generate anintermediate image by using the image data of adjacent frames as in thefirst method “frame (field) interpolation based on motion compensation”;it is possible to generate a subframe image with a small amount ofcalculation as in other embodiments.

Fourth Embodiment

In each embodiment described above, the respective units constitutingthe apparatus shown in FIG. 1 or 12 are implemented by hardware.However, the respective units constituting the apparatus shown in FIG. 1or 12 are implemented by software. In this case, the software is held invarious types of storage devices which a computer has. When the CPUexecutes this software, the computer implements the function of eachunit shown in FIG. 1 or 12.

FIG. 16 is a block diagram showing the hardware arrangement of thecomputer to which this embodiment can be applied. A computer 1601 is ageneral-purpose information processing apparatus such as a personalcomputer in widespread use. In the computer 1601, the respective blocksto be described later are connected to each other via a bus 1607 and canexchange various data.

Depending on the apparatus to which the computer 1601 is applied, notall the constituent elements shown in FIG. 16 are necessary. Therefore,some of the constituent elements shown in FIG. 16 can be omitted asneeded. In addition, the constituent elements shown in FIG. 16 may bereplaced by hardware having equivalent functions. Furthermore, thecomputer 1601 may be constituted by a plurality of computer units.

Referring to FIG. 16, reference numeral 1602 denotes a CPU, whichcontrols the overall computer 1601 by using computer programs and dataloaded in a main memory 1603. The CPU 1602 executes each processingdescribed above which is performed by an image processing apparatus towhich the computer 1601 is applied. The main memory 1603 is typically aRAM. The main memory 1603 has an area for temporarily storing programsand data loaded from various types of storage devices. The storagedevices include an HDD (Hard Disk Drive) 1604, a CD drive 1609, a DVDdrive 1610, and an FDD (Floppy® Disk Drive) 1611. The main memory 1603further has an area for temporarily storing image data acquired from ascanner 1617 via an I/F (Interface) 1615. In addition, the main memory1603 has a work area which is used by the CPU 1602 to execute variouskinds of processes. The main memory 1603 can provide various kinds ofinformation recording locations including the above areas, as needed.

The HDD 1604 holds an OS (Operating System), various kinds of images(including document images), and the like. The HDD 1604 holds programsand data which make the CPU 1602 control the functions of the respectiveunits shown in FIG. 16. The HDD 1604 also holds programs and data whichmake the CPU 1602 execute each processing described above which isexecuted by the apparatus to which the computer 1601 is applied. Theprograms and data held in the HDD 1604 are loaded into the main memory1603 under the control of the CPU 1602, as needed, and are processed bythe CPU 1602. Note that the HDD 1604 may hold some of the pieces ofinformation described as those stored in the main memory 1603.

Reference numeral 1605 denotes a video controller. The video controller1605 transmits display data such as image data and character datareceived from the main memory 1603, the HDD 1604, and the like assignals to a monitor 1606. The monitor 1606 includes a CRT and a liquidcrystal display. The monitor 1606 displays images, characters, and thelike based on signals received from the video controller 1605.

Reference numeral 1608 denotes an I/F for connecting a printer 1616 tothe computer 1601. The computer 1601 transmits print data to the printer1616 via the I/F 1608. The computer 1601 can also receive the stateinformation of the printer 1616, transmitted by the printer 1616, viathe I/F 1608. Reference numeral 1609 denotes a CD drive, which reads outprograms and data recorded on CDs as recording media. The CD drive 1609also transmits readout programs and data to the HDD 1604, the mainmemory 1603, and the like.

Reference numeral 1610 denotes a DVD drive, which reads out programs anddata recorded on DVDs as recording media. The DVD drive 1610 alsotransmits readout programs and data to the HDD 1604, the main memory1603, and the like. Reference numeral 1611 denotes an FDD, which readsout programs and data recorded on Floppy® disks as recording media. TheFDD 1611 also transmits readout programs and data to the HDD 1604, themain memory 1603, and the like.

Reference numerals 1613 and 1614 denote a mouse and a keyboard,respectively, as operation input devices. The user of the computer 1601can input various kinds of instructions to the CPU 1602 by operating themouse 1613 and the keyboard 1614. Reference numeral 1612 denotes an I/Ffor connecting the keyboard 1614 and the mouse 1613 to the bus 1607. Theoperation instructions input by the user via the mouse 1613 and thekeyboard 1614 are transmitted as signals to the CPU 1602 via the I/F1612.

Reference numeral 1615 denotes an I/F for connecting the scanner 1617,which generates image data by reading documents, films, and the like, tothe computer 1601. The image data generated by the scanner 1617 istransmitted to the HDD 1604, the main memory 1603, and the like via theI/F 1615. Reference numeral 1618 denotes an I/F for exchanginginformation with electronic devices such as other computers. Informationincluding image data acquired from a network 1619 in response to aninstruction from the CPU 1602 is transmitted to the HDD 1604, the mainmemory 1603, and the like via the I/F 1618.

Other Embodiments

The four embodiments have been described above. However, the aboveembodiments can be variously changed within the spirit and scope of thepresent invention. Although each filter function has been described as afunction based on a Gaussian function, any type of filter function maybe used. It is possible to acquire filter coefficients by translating afilter function as a base in accordance with equations (4) orinequalities (5).

As has been described above, the image display apparatus according tothe present invention can reduce an image lag in an area with motionwhich occurs in the second method “spatial frequency splitting”. It isnot necessary to generate an intermediate image by using the image dataof adjacent frames as in the first method “frame (field) interpolationbased on motion compensation”; it is possible to generate a subframeimage with a small amount of calculation.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (for example, computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2009-132424, filed Jun. 1, 2009 which is hereby incorporated byreference herein in its entirety.

1. An image processing apparatus which processes a moving imagecomprising successive frame images, comprising: a unit configured toobtain a frame image of interest as an input image, and to obtain aframe image subsequent to the frame image of interest as a subsequentimage; a unit configured to obtain N replicated images from the inputimage; a generating unit configured to generate a low-frequency enhancedimage whose low-frequency component is enhanced, from the replicatedimage by using a filter for enhancing a low-frequency component; asubtraction unit configured to generate a high-frequency enhanced imagewhose high-frequency component is enhanced, from the low-frequencyenhanced image and the replicated image; a synthesizing unit configuredto generate a high-frequency output image by synthesizing thehigh-frequency enhanced image and the low-frequency enhanced image at apredetermined ratio; and an output unit configured to select and outputone of the low-frequency enhanced image and the high-frequency outputimage in place of each of the replicated images, said generating unitcomprising a unit configured to obtain a motion vector of an objectdepicted in the input image and the subsequent image, and a filter unitconfigured to generate the low-frequency enhanced image, when processinga pixel at a pixel position P in the replicated image, by performing,for each pixel position in the replicated image, a process of specifyinga pixel position Q spaced apart from the pixel position P in thereplicated image by a predetermined distance in an opposite direction tothe motion vector, a process of applying the filter to pixels around thespecified pixel position Q in the replicated image, to obtain a pixelvalue at the pixel position P in the low-frequency enhanced image. 2.The apparatus according to claim 1, wherein said subtraction unitgenerates, as the high-frequency enhanced image, an image obtained bysubtracting each pixel value in the low-frequency enhanced image from acorresponding pixel value of the input image, and when said output unitoutputs a high-frequency output images and the (N−a) low-frequencyenhanced images for one input image, said synthesizing unit generates,as the high-frequency output image, an image obtained by adding a valueobtained by multiplying each pixel value of the high-frequency enhancedimage by (N−a) to a corresponding pixel value of the low-frequencyenhanced image.
 3. The apparatus according to claim 1, wherein whenprocessing an i:th replicated image for the input image, said filterunit acquires a motion vector indicating a motion of the pixel positionP, acquires, as a moving amount, a vector having a value obtained bymultiplying a value of the motion vector by (−(i−1)/N), and specifies aposition moved from the pixel position P by the moving amount as thepixel position Q.
 4. The apparatus according to claim 3, wherein saidfilter unit acquires values, which are obtained by applying the filterto pixels around the pixel position Q, by applying filter coefficientsto pixels around the pixel position P, which filter coefficients areobtained by moving a filter function, which provides coefficient valuesof the filter, by the moving amount.
 5. An image processing apparatuswhich processes a moving image comprising successive frame images,comprising: a unit configured to obtain a frame image of interest as aninput image, and to obtain a frame image subsequent to the frame imageof interest as a subsequent image; a unit configured to obtain Nreplicated images from the input image; an alternative generating unitconfigured to generate a high-frequency enhanced image whosehigh-frequency component is enhanced, from the replicated image by usinga filter for enhancing a high-frequency component; a unit configured togenerate a low-frequency enhanced image whose low-frequency component isenhanced, from the high-frequency enhanced image and the replicatedimage; a unit configured to generate a high-frequency output image bysynthesizing the high-frequency enhanced image and the low-frequencyenhanced image at a predetermined ratio; and a unit configured to selectand output one of the low-frequency enhanced image and thehigh-frequency output image in place of each of the replicated images,said alternative generating unit comprising a unit configured to obtaina motion vector of an object depicted in the input image and thesubsequent image, and a unit configured to generate the high-frequencyenhanced image, when processing a pixel at a pixel position P in thereplicated image, by performing, for each pixel position in thereplicated image, a process of specifying a pixel position Q spacedapart from the pixel position P in the replicated image by apredetermined distance in an opposite direction to the motion vector, aprocess of applying the filter to pixels around the specified pixelposition Q in the replicated image, to obtain a pixel value at the pixelposition P in the high-frequency enhanced image.
 6. An image processingmethod executed by an image processing apparatus comprising a unitconfigured to process a moving image comprising successive frame images,comprising: a step of obtaining a frame image of interest as an inputimage, and obtaining a frame image subsequent to the frame image ofinterest as a subsequent image; a step of obtaining N replicated imagesfrom the input image; a generating step of generating a low-frequencyenhanced image whose low-frequency component is enhanced, from thereplicated image by using a filter for enhancing a low-frequencycomponent; a step of generating a high-frequency enhanced image whosehigh-frequency component is enhanced, from the low-frequency enhancedimage and the replicated image; a step of generating a high-frequencyoutput image by synthesizing the high-frequency enhanced image and thelow-frequency enhanced image at a predetermined ratio; and an outputstep of selecting and outputting one of the low-frequency enhanced imageand the high-frequency output image in place of each of the replicatedimages, the generating step comprising a step of obtaining a motionvector of an object depicted in the input image and the subsequentimage, and a filter step of generating the low-frequency enhanced image,when processing a pixel at a pixel position P in the replicated image,by performing, for each pixel position in the replicated image, aprocess of specifying a pixel position Q spaced apart from the pixelposition P in the replicated image by a predetermined distance in anopposite direction to the motion vector, a process of applying thefilter to pixels around the specified pixel position Q in the replicatedimage, to obtain a pixel value at the pixel position P in thelow-frequency enhanced image.
 7. An image processing method executed byan image processing apparatus comprising a unit configured to process amoving image comprising successive frame images, comprising: a step ofobtaining a frame image of interest as an input image, and obtaining aframe image subsequent to the frame image of interest as a subsequentimage; a step of obtaining N replicated images from the input image; analternative generating step of generating a high-frequency enhancedimage whose high-frequency component is enhanced, from the replicatedimage by using a filter for enhancing a high-frequency component; a stepof generating a low-frequency enhanced image whose low-frequencycomponent is enhanced, from the high-frequency enhanced image and thereplicated image; a step of generating a high-frequency output image bysynthesizing the high-frequency enhanced image and the low-frequencyenhanced image at a predetermined ratio; and a step of selecting andoutputting one of the low-frequency enhanced image and thehigh-frequency output image in place of each of the replicated images,the alternative generating step comprising a step of obtaining a motionvector of an object depicted in the input image and the subsequentimage, and a step of generating the high-frequency enhanced image, whenprocessing a pixel at a pixel position P in the replicated image, byperforming, for each pixel position in the replicated image, a processof specifying a pixel position Q spaced apart from the pixel position Pin the replicated image by a predetermined distance in an oppositedirection to the motion vector, a process of applying the filter topixels around the specified pixel position Q in the replicated image, toobtain a pixel value at the pixel position P in the high-frequencyenhanced image.
 8. A non-transitory computer-readable storage medium,which stores a computer program for causing a computer to function aseach unit of an image processing apparatus defined in claim
 1. 9. Anon-transitory computer-readable storage medium, which stores a computerprogram for causing a computer to function as each unit of an imageprocessing apparatus defined in claim 5.